Papers with probing language models
Linguistically Grounded Analysis of Language Models using Shapley Head Values (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for probing language models for morphosyntactic constructions are not well understood . language models gain knowledge of grammatical phenomena during pretraining, but exactly how this knowledge is encoded is not well established. |
| Approach: | They propose a method for probing language models via Shapley Head Values . they use a BLiMP dataset to test their method on linguistic constructions based on a Shaply Head Value method . |
| Outcome: | The proposed method can be used to investigate linguistic knowledge in language models . it shows that attention heads responsible for processing related linguistic phenomena cluster together . |
Social Bias Probing: Fairness Benchmarking for Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for evaluating social biases in language models have been limited to binary association tests on small datasets. |
| Approach: | They propose a framework for probing language models for social biases by assessing disparate treatment . they use a large-scale benchmark to examine the diversity of identities and stereotypes . |
| Outcome: | The proposed framework expands the analysis beyond the binary comparison of stereotypical versus anti-stereotypical identities to include a diverse range of identities and stereotypes. |